Effective load balancing for cluster-based servers employing job preemption

A cluster-based server consists of a front-end dispatcher and multiple back-end servers. The dispatcher receives incoming jobs, and then decides how to assign them to back-end servers, which in turn serve the jobs according to some discipline. Clusterbased servers have been widely deployed, as they combine good performance with low costs. Several assignment policies have been proposed for cluster-based servers, most of which aim to balance the load among back-end servers. There are two main strategies for load balancing: The first aims to balance the amount of work at back-end servers, while the second aims to balance the number of jobs assigned to back-end servers. Examples of policies using these strategies are Dynamic and LC (Least Connected), respectively. In this paper we propose a policy, called LC*, which combines the two aforementioned strategies. The paper shows experimentally that when preemption is admitted (i.e., when jobs execute concurrently on back-end servers), LC* substantially outperforms both Dynamic and LC in terms of response time metrics. This improved performance is achieved by using only information readily available to the dispatcher, rendering LC* a practical policy to implement. Finally, we study a refinement, called ALC* (Adaptive LC*), which further improves on the response-time performance of LC* by adapting its actions to incoming traffic rates.